Anomaly detection for non-stationary data

ABSTRACT

A method of detecting anomalies in a time series is disclosed. A training time series corresponding to a process is extracted from an initial time series corresponding to the process, the training time series including a subset of the initial time series. Outlier data points in the training time series are modified based on predetermined acceptability criteria. A plurality of prediction methods are trained using the training time series. An actual data point corresponding to the initial time series is received. The plurality of prediction methods are used to determine a set of predicted data points corresponding to the actual data point. It is determined whether the actual data point is anomalous based on a calculation of whether each of the set of predicted data points is statistically different from the actual data point.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/588,355, filed on Dec. 31, 2014, which is incorporated herein byreference in its entirety.

TECHNICAL FIELD

The present application relates generally to the technical field of dataprocessing, and, in one specific example, to detecting anomalies in timeseries data corresponding to processes executing on one or more servers.

BACKGROUND

Various computer systems, such as electronic commerce (e-commerce)systems or network-based publication systems (e.g., eBay, Amazon, orCraigslist), which facilitate exchanges of items (e.g., goods orservices) between users, may include many (e.g., millions or billions)of processes executing on one or more servers. Each of these processesmay have one or more data inputs and one or more data outputs.Furthermore, input and output data in such systems may be non-stationarydata. For example, data generated by a set of processes for a firstlocale may have statistical properties that vary from data generated bythe same set of processes for a second locale. Given the complexity ofsuch systems, determining their overall health may be well outside thepurview of human operators.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings.

FIG. 1 is a network diagram depicting a client-server system withinwhich various example embodiments may be deployed.

FIG. 2 is a block diagram illustrating multiple server applicationsthat, in various example embodiments, are provided as part of thenetworked system of FIG. 1.

FIG. 3 is an example embodiment of a method of determining a health of acomplex system.

FIG. 4 is an example embodiment of a method of detecting anomalies withrespect to a computer process executing within a complex system.

FIG. 5 is a block diagram illustrating an example anomaly detectiondaily process.

FIG. 6 is an example user interface of a graphical user interface forrepresenting a health of particular computer process executing within acomplex system.

FIG. 7 is a block diagram of machine in the example form of a computersystem within which instructions for causing the machine to perform anyone or more of the methodologies discussed herein may be executed.

DETAILED DESCRIPTION

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide an understanding ofvarious embodiments of the present subject matter. It will be evident,however, to those skilled in the art that various embodiments may bepracticed without these specific details.

An electronic commerce system may be a complex system havinginput-output behavior that is difficult to quantify. However, theoverall health of the system can be quantified at least in the sensethat the overall health is reflected by the output of the system'sdata-generating processes. In other words, the health of a complexsystem may be reflected by the data it generates. For example, variousfacets of system health may be reflected by data (e.g., time seriesdata) corresponding to various data categories, such as the financial,behavioral, clickstream-derived, and server-derived categories. Forexample, it is not hard to imagine that broad-impact events, such as achange to Google's SEO algorithm, may affect data in many, if not all,categories of data of an electronic commerce system. On the other hand,events with a narrower impact, such as code bugs or targetedcyber-attacks, may only affect data belonging to a specific category ofdata. Regardless of the scope of the impact, anomalies in system healthare manifested in data. Such anomalies may be of a positive(business-enhancing) or negative (business-degrading) nature.

Detecting anomalies in system health may be the first step towardsattenuating their negative business impact, stimulating their positivebusiness impact, and understanding their underlying root causes. To thisend, an anomaly detection system may continuously monitor data producedby each of the system's data-generating processes.

In some embodiments, an e-commerce system has billions ofdata-generating processes. Working with such a large quantity of datasets may be outside of the purview of human operators, which raises theneed for automation. Further, any process that continuously monitorsbillions of data sets should have an implementation that is massivelyscalable. More subtly, the data-generating processes themselves mayproduce data with diverse characteristics; this issue raises the needfor anomaly detection algorithms which are data-adaptive. Thus, theback-end of an anomaly-detection system must be driven by an anomalydetection algorithm that is automated, massively scalable, anddata-adaptive.

To make the notion of an anomaly more precise, we first define what itmeans for the system to be in normal health. To this end, we make thefollowing operational definition: The system is in normal health if eachof its data-generating processes is in a state of statisticalequilibrium. Under this definition, an anomaly is a deviation of one ofthe system's data-generating processes from statistical equilibrium. Inpractice, an anomaly in a particular data-generating process will beevident from the data it generates.

As indicated above, the anomaly-detection system should bedata-adaptive. A challenge to devising such a system is that thedata-adaptation may need to occur within a given data set, not justacross multiple data sets. For example, a given data set may havestatistical properties that vary across local time regimes. In thiscase, we say that the data (and its corresponding data-generatingprocess) is non-stationary. Non-stationarity complicates our approach toanomaly detection, since the notion of “statistical equilibrium” mustalso be time-adaptive. The anomaly-detection system is designed toseamlessly handle non-stationarity by integrating data from multiplelocal time regimes.

In various embodiments, the anomaly-detection system is, fundamentally,a statistical test for anomalies. In the statistical realm, there mustbe a threshold set that describes the amount of expected false anomalydetections. The anomaly-detection system is designed to minimize thenumber of expected false anomaly detections (e.g., by combining datafrom a battery of different prediction algorithms).

A method of detecting anomalies in a time series is disclosed. Atraining time series corresponding to a process is extracted from theinitial time series. Outlying data points in the training time seriesare modified based on predetermined acceptability criteria. A pluralityof prediction methods are trained using the training time series andpredictions (e.g., one-step-ahead predictions) are obtained using theestimated parameters for each prediction method. Prediction errors arethen computed. It is determined whether a data point is anomalous basedon a determination of whether the corresponding prediction error isstatistically different than the immediate prediction error history of agiven length relative to that data point. The final determination isbased on a combination of the information from the plurality ofprediction methods.

This method and various embodiments disclosed herein may be implementedas a computer system having one or more modules (e.g., hardware modulesor software modules). This method and various embodiments disclosedherein may be embodied as instructions stored on a machine-readablemedium that, when executed by a processor, cause the processor toperform the method.

FIG. 1 is a network diagram depicting a system 100 within which variousexample embodiments may be deployed. A networked system 102, in theexample forms of a network-based marketplace or other publicationsystem, provides server-side functionality, via a network 104 (e.g., theInternet or Wide Area Network (WAN)) to one or more clients machines110. FIG. 1 illustrates client application(s) 112 on the client machines110. Examples of client application(s) 112 may include a web browserapplication, such as the Internet Explorer browser developed byMicrosoft Corporation of Redmond, Washington or other applicationsupported by an operating system of the device, such as Windows, iOS orAndroid operating systems. Each of the client application(s) 112 mayinclude a software application module (e.g., a plug-in, add-in, ormacro) that adds a specific service or feature to a larger system.

An API server 114 and a web server 116 are coupled to, and provideprogrammatic and web interfaces respectively to, one or more applicationservers 118. The application servers 118 host one or more serverapplication(s) 120. The application servers 118 are, in turn, shown tobe coupled to one or more database servers 124 that facilitate access toone or more databases 126 or data stores.

The applications 120 may provide a number of marketplace functions andservices to users that access the networked system 102. While theapplications 120 are shown in FIG. 1 to form part of the networkedsystem 102, in alternative embodiments, the various applications 120 mayform part of a service that is separate and distinct from the networkedsystem 102.

Further, while the system 100 shown in FIG. 1 employs a client-serverarchitecture, various embodiments are, of course, not limited to such anarchitecture, and could equally well find application in a distributed,or peer-to-peer, architecture system, for example. The various serverapplications 120 could also be implemented as standalone softwareprograms, which do not necessarily have networking capabilities.Additionally, although FIG. 1 depicts machines 110 as being coupled to asingle networked system 102, it will be readily apparent to one skilledin the art that client machines 110, as well as client applications 112,may be coupled to multiple networked systems, such as paymentapplications associated with multiple payment processors or acquiringbanks (e.g., PayPal, Visa, MasterCard, and American Express).

Web applications executing on the client machine(s) 110 may access thevarious applications 120 via the web interface supported by the webserver 116. Similarly, native applications executing on the clientmachine(s) 110 may access the various services and functions provided bythe applications 120 via the programmatic interface provided by the APIserver 114. An example of one of the client application(s) 112 may be aseller application (e.g., the TurboLister application developed by eBayInc., of San Jose, Calif.) that enables sellers to author and managelistings on the networked system 102 in an off-line manner, and toperform batch-mode communications between the client application(s) 112and the networked system 102. Other examples of client application(s)112 may be third-party applications. For example, the third-partyapplications may, utilizing information retrieved from the networkedsystem 102, support one or more features or functions on a websitehosted by the third party. The third-party website may, for example,provide one or more promotional, marketplace or payment functions thatare supported by the relevant applications of the networked system 102.

FIG. 2 is a block diagram illustrating multiple server applications 120that, in various example embodiments, are provided as part of thenetworked system 102. The server applications 120 may be hosted ondedicated or shared server machines (not shown) that are communicativelycoupled to enable communications between server machines. The serverapplications 120 themselves are communicatively coupled (e.g., viaappropriate interfaces) to each other and to various data sources, so asto allow information to be passed between the server applications 120and thereby to share and access common data. The server applications 120may furthermore access one or more databases 126 via the databaseservers 124.

The networked system 102 may provide a number of publishing, listing andprice-setting mechanisms whereby a seller may list (or publishinformation concerning) goods or services for sale, a buyer can expressinterest in or indicate a desire to purchase such goods or services, anda price can be set for a transaction pertaining to the goods orservices. To this end, the server applications 120 are shown to includeat least one publication application 200 and one or more auctionapplications 202 which support auction-format listing and price settingmechanisms (e.g., English, Dutch, Vickrey, Chinese, Double, Reverseauctions etc.). The various auction applications 202 may also provide anumber of features in support of such auction-format listings, such as areserve price feature whereby a seller may specify a reserve price inconnection with a listing and a proxy-bidding feature whereby a biddermay invoke automated proxy bidding.

A number of fixed-price applications 204 support fixed-price listingformats (e.g., the traditional classified advertisement-type listing ora catalogue listing) and buyout-type listings. Specifically, buyout-typelistings (e.g., including the Buy-It-Now (BIN) technology developed byeBay Inc., of San Jose, Calif.) may be offered in conjunction withauction-format listings, and allow a buyer to purchase goods orservices, which are also being offered for sale via an auction, for afixed-price that is typically higher than the starting price of theauction.

Store applications 206 allow a seller to group listings within a“virtual” store, which may be branded and otherwise personalized by andfor the seller. Such a virtual store may also offer promotions,incentives and features that are specific and personalized to a relevantseller.

Reputation applications 208 allow users that transact, utilizing thenetworked system 102, to establish, build and maintain reputations,which may be made available and published to potential trading partners.Consider that where, for example, the networked system 102 supportsperson-to-person trading, users may otherwise have no history or otherreference information whereby the trustworthiness and credibility ofpotential trading partners may be assessed. The reputation applications208 allow a user (e.g., through feedback provided by other transactionpartners) to establish a reputation within the networked system 102 overtime. Other potential trading partners may then reference such areputation for the purposes of assessing credibility andtrustworthiness.

Personalization applications 210 allow users of the networked system 102to personalize various aspects of their interactions with the networkedsystem 102. For example a user may, utilizing an appropriatepersonalization application 210, create a personalized reference page atwhich information regarding transactions to which the user is (or hasbeen) a party may be viewed. Further, a personalization application 210may enable a user to personalize listings and other aspects of theirinteractions with the networked system 102 and other parties.

The networked system 102 may support a number of marketplaces that arecustomized, for example, for specific geographic regions. For instance,a version of the networked system 102 may be customized for the UnitedKingdom, whereas another version of the networked system 102 may becustomized for the United States. Each of these versions may operate asan independent marketplace, or may be customized (or internationalized)presentations of a common underlying marketplace. The networked system102 may accordingly include a number of internationalizationapplications 212 that customize information (and/or the presentation ofinformation) by the networked system 102 according to predeterminedcriteria (e.g., geographic, demographic or marketplace criteria). Forexample, the internationalization applications 212 may be used tosupport the customization of information for a number of regionalwebsites that are operated by the networked system 102 and that areaccessible via respective web servers 116.

Navigation of the networked system 102 may be facilitated by one or morenavigation applications 214. For example, a search application (as anexample of a navigation application) may enable keyword searches oflistings published via the networked system 102. A browse applicationmay allow users to browse various category, catalogue, or inventory datastructures according to which listings may be classified within thenetworked system 102. Various other navigation applications may beprovided to supplement the search and browsing applications.

In order to make listings available via the networked system 102 asvisually informing and attractive as possible, the marketplaceapplications 120 may include one or more imaging applications 216, whichusers may utilize to upload images for inclusion within listings. Animaging application 216 also operates to incorporate images withinviewed listings. The imaging applications 216 may also support one ormore promotional features, such as image galleries that are presented topotential buyers. For example, sellers may pay an additional fee to havean image included within a gallery of images for promoted items.

Listing creation applications 218 allow sellers to conveniently authorlistings pertaining to goods or services that they wish to transact viathe networked system 102, and listing management applications 220 allowsellers to manage such listings. Specifically, where a particular orpredetermined seller has authored and/or published a large number oflistings, the management of such listings may present a challenge. Thelisting management applications 220 provide a number of features (e.g.,auto-relisting, inventory level monitors, etc.) to assist the seller inmanaging such listings. The listing creation application 218 and listingmanagement applications 220 may allow sellers to manage listing in bulk(e.g., in a single operation, such as by an uploading of a file) andprovide templates for sellers to manage category-specific,vendor-specific, or general-type-specific (e.g., catalog or ticket)listings. One or more post-listing management applications 222 alsoassist sellers with a number of activities that typically occurpost-listing. For example, upon completion of an auction facilitated byone or more auction applications 202, a seller may wish to leavefeedback regarding a particular or predetermined buyer. To this end, apost-listing management application 222 may provide an interface to oneor more reputation applications 208, so as to allow the seller toconveniently provide feedback regarding multiple buyers to thereputation applications 208.

Dispute resolution applications 224 provide mechanisms whereby disputesarising between transacting parties may be resolved. For example, thedispute resolution applications 224 may provide guided procedureswhereby the parties are guided through a number of operations in anattempt to settle a dispute. In the event that the dispute cannot besettled via the guided procedures, the dispute may be escalated to athird-party mediator or arbitrator.

A number of fraud prevention applications 226 implement fraud detectionand prevention mechanisms to reduce the occurrence of fraud within thenetworked system 102.

Messaging applications 228 are responsible for the generation anddelivery of messages to users of the networked system 102. Thesemessages may, for example, advise users regarding the status of listingsat the networked system 102 (e.g., providing “outbid” notices to biddersduring an auction process or providing promotional and merchandisinginformation to users). Respective messaging applications 228 may utilizeany one of a number of message delivery networks and platforms todeliver messages to users. For example, messaging applications 228 maydeliver electronic mail (e-mail), instant message (IM), Short MessageService (SMS), text, facsimile, or voice (e.g., Voice over IP (VoIP))messages via the wired (e.g., the Internet), Plain Old Telephone Service(POTS), or wireless (e.g., mobile, cellular, WiFi, WiMAX) networks.

Merchandising applications 230 support various merchandising functionsthat are made available to sellers to enable sellers to increase salesvia the networked system 102. The merchandising applications 230 alsooperate the various merchandising features that may be invoked bysellers, and may monitor and track the success of merchandisingstrategies employed by sellers.

The networked system 102 itself, or one or more parties that transactvia the networked system 102, may operate loyalty programs that aresupported by one or more loyalty/promotion applications 232. Forexample, a buyer may earn loyalty or promotions points for eachtransaction established and/or concluded with a particular orpredetermined seller, and may be offered a reward for which accumulatedloyalty points can be redeemed.

Each of the server application(s) 120 may be associated with one or morecomputer processes, each having one or more sets of output data. One ormore anomaly detection application(s) 152 executing on or more anomalydetection machine(s) 150 may be configured to detect anomalies in thesystem 100, as described in more detail below.

FIG. 3 is an example embodiment of a method 300 of determining a healthof a system, such as the system 100 of FIG. 1. In various embodiments,the method 300 may be implemented by one or more of the anomalydetection application(s) 152. In various embodiments, the methodimplements a framework for detecting anomalies in time series data.

In various embodiments, the method 300 may process one or a multitude oftime series such that it processes one of such time series at a time forone time point. The method 300 can then be repeated for every time pointwithin a time duration of interest, in order, from the farthest timepoint to the latest. While processing a time series S (e.g., an initialtime series) for time point t, at 302, the preparatory stage isconcerned with data preprocessing. A training time series correspondingto a predetermined or particular data-generating process is receivedsuch that this training time series, say T, of length n where n issmaller than t, contains data elements from S corresponding to timepoints from (t−n) to (t−1) in the initial time series. In variousembodiments, the training time series may be further modified byimplementing a process to ensure that the time series T adheres tospecific predetermined standards.

At operation 304, the training stage operates using data contained inthe time series T. In various embodiments, this stage selects locallyoptimal parameters for each of a prescribed set of prediction methods(e.g., one-step prediction methods). In this way, the training stage isused to assess certain statistical properties for each member of the setof the prediction methods specifically based on the time series Tselected during operation 302 corresponding to the time point t ofinterest.

At operation 306, the prediction stage functions use the locally optimalstatistical properties generated in operation 304 for each member of theset of prediction methods to generate predictions (e.g., one-step aheadpredictions) for each member of the set of prediction methodscorresponding to time point t for the time series S.

At operation 308, the anomaly detection stage identifies a potentialanomaly in time series S at time point t. The actual (measured) data attime point t is compared with the predictions from operation 306 and astatistical test is applied to the differences between the predicted andactual data. The results of this test are used to flag the data at timepoint t as anomalous or not. Overall, this procedure is consistent withthe intuitive notion that anomalies are “unpredictable” and it is ableto identify when the underlying data-generating processes are operatingoutside of their usual state.

FIG. 4 is an example embodiment of a method 400 of detecting anomalieswith respect to a complex system, such as the system 100 of FIG. 1. Invarious embodiments, the method 400 may be implemented by one or more ofthe anomaly detection application(s) 152.

In this embodiment, the method 400 is implemented such that it isexecuted for each time series of interest and for the entire timeduration of interest, one time point at a time. For example, supposethat the data given is a time series S, indexed from time point 1 to M,and comprised of observations X(1), X(2), . . . , X(M). Also, assume thetime duration of interest is from time point P to time point M (theend). So, the anomaly detection process starts with time point P.

At operation 402, a training time series T of training length N isextracted from S such that it contains data elements from time point(P−N) to time point (P−1). In various embodiments, it is assumed that Nis significantly less than M.

At operation 404, outlying data points in the time series T are capped.For example, data pre-processing operations may be applied to thetraining time series to bring identified outliers to within the valuesof other data values.

At operation 406, the training time series T is used to generate theparameters for each of a plurality of prediction methods. For example,the following steps may be performed: using the training time series Tto estimate the parameters of each of q different pre-selectedprediction methods (e.g., to ensure that each of the prediction methodsis trained to use locally-optimal parameters); and applying the qdifferent prediction methods obtain q different prediction values (e.g.,one-step ahead prediction values): Y(P, 1), Y(P, 2), . . . , Y(P, q).

Thus, q prediction values may be produced for time point P. From theseprediction values, q prediction errors may be computed such that: E(P,j)=Y(P, j)−X(P) where j varies from 1 to q. In various embodiments,these prediction errors may quantify the deviations of the predictionvalues from their corresponding measured values within S.

Thus, in various embodiments, it is assumed that each of the predictionmethods produces one step-ahead prediction errors which are independentand normally distributed, when regarded as random variables. In variousembodiments, parameters of the normal distribution may also bedetermined, including the mean and variance. Furthermore, in variousembodiments, the normal distribution and associated parameters may bedetermined for each of a number of preselected prediction methods.

At operation 408, q prediction error time series of length L (for timepoints from (P−L) to (P−1) where P is smaller than L) corresponding tothe q prediction methods applied in the context on time series S in thepast, are extracted. These prediction error time series are the resultof the operation 400 for historical time points, i.e., time points olderthan P.

At operation 410, it is determined whether data point at time point P isanomalous. For example, for a specific prediction method (identified byindex j), the prediction errors E(P−L, j), E(P−1, j) computed in thepast comprise L samples of a normally distributed random variable withmean μ(j) and standard deviation σ(j). A multivariate normal jointprobability distribution is then fitted with mean μ=(μ(1), . . . μ(j))and standard deviation σ=(σ(1), . . . , σ(j)) assuming the q differentprediction methods produce independent prediction errors. A distancebetween each E(P, j) and the fitted multivariate normal jointprobability distribution is measured (e.g., using the Mahalanobisdistance). If the distances computed exceed a predetermined threshold,the time index P is marked as containing an anomaly.

At operation 412, operations 402 to 410 may then be repeated from timeperiods P to M.

At operation 414, a user interface facilitates data exploration andprovides a visual indication of whether the data point of interest inthe time series S is anomalous. For example, as shown in FIG. 6, thetime series data corresponding to a process is presented in a userinterface and anomalous data points in the data series are flagged.

FIG. 5 is a block diagram illustrating an example anomaly detectiondaily process 500 (the time interval between consecutive time points inthis specific embodiment is 1 day). In various embodiments, the anomalydetection daily process 500 may be implemented daily by one or more ofthe anomaly detection application(s) 152. In this specific embodiment,the process is built to: process millions of time series; accept a datafeed from an external data source and append the incremental daily data(pertaining to the latest date for which the time series data areavailable) to the data source within the embodiment (system); processall the time series and detect whether the latest date appended hasanomalies and for which time series. At operation 502, ascheduler/controller application manages (1) kicking off a first process504 (e.g., a database process) to ingest incremental data and (2)kicking off a second process 514 (e.g., a statistical process) toprocess the incremental data for anomaly detection.

At operation 506, one or more database tables are accessed or updatedwith the latest update of incremental daily data. The accessed orupdated tables may include raw data , output data already processed bythe statistical process at an earlier time period or other (e.g.,configuration) information. As a result of this update, all the timeseries in the embodiment are appended with their respective dataelements corresponding to the latest date (the date being processed).These time series are further grouped together into logical units ofwork (LUWs). A LUW can be defined as a group of individual time seriescombined together based on some predetermined criteria to be processedtogether. The criteria could be related to the logic stemming from thenature of the data or it be could a random grouping purely for thepurpose of streamlining the amount of data and work passed on to thestatistical engine in one request.

At operation 508, according to a first procedure (e.g., stored procedureA), a list of logical units of work (LUWs) are retrieved.

At operation 510, according to a second procedure (e.g., storedprocedure B), a first set of data is retrieved which contains the datafor all the time series which are a part of a specific LUW (e.g., a3-year output data history up until yesterday for all time series in agiven logical unit of work (LUW)).

At operation 512, according to a third procedure (e.g., stored procedureC), a different set of data is retrieved for all the time series in aspecific LUW (e.g., an output data history for 1 year up until yesterdayfor all time series in a given LUW). The set of data also includes theprocessed information written at a previous time point for the timeseries in the LUW of interest by operation 514.

At operation 514, a master program in the statistical engine manages theexecution of the anomaly detection procedures for all LUWs. The masterprogram in operation 514 distributes the processing of LUWs across alarge number of worker programs such that each worker program processes1 LUW and runs operation 516 for that LUW independently on an individualprocessing thread or processing core. The worker program moves on toprocess the next LUW only once it has finished processing the first LUW.

At operation 516, which is each executed independently on a separateprocessing thread or processing core, for a given LUW, the workerprogram(s) receives LUW actuals data (e.g., using stored procedure B,operation 510). Additionally, for each LUW, the worker program(s) runpredictions, such as (a) multiple linear regression-based (MLR-based)one-day-ahead prediction for today, (b) MLR-based two-day-aheadprediction for tomorrow, (c) MLR-based one-day-ahead prediction fortoday using seven-day moving average (MA), (d) a proprietary predictionmethod, such as EMD-spectral prediction (ESP)-based one-day-aheadprediction for today, (c) or other prediction methods, such as dynamiclinear model (DLM), wavelet-based model, empirical mode decomposition(EMD)-based model, or non-stationary ESP model.

Furthermore, the worker program(s) receive previously processed output(prediction and residual) history (e.g., 1 year of residuals) usingstored procedure C (operation 512), calculates residuals and anomalydetection related metrics for yesterday, and, at 518, write backprocessed information for yesterday, today, and tomorrow to the outputtable using stored procedure D. A graphical user interface may then begenerated to visually present the information in the output table (e.g.,see FIG. 6).

FIG. 6 is an example user interface 600 of a graphical user interfacefor representing a health of particular process executing within acomplex system, such as the system 100 of FIG. 1. In variousembodiments, data points of a time series corresponding to a selectedprocess may be presented in graph. Here, the data points correspond tothe gross merchandise volume (GMV) in an e-commerce system in the U.S.for products in an electronics/cell phones & accessories category. Aone-day MLR model of data points is displayed over a sixth-month timeperiod. Anomalous data points are flagged (e.g., with a graphicalindicator corresponding to the position of the data point within thetime series). In various embodiments, the identification of theanomalous data points is based on an analysis of the data points withrespect to a combination of a plurality of data models (e.g., the 1-dayMLR model, the one-day MLR model on seven-day MA, the two-day MLR model,and a proprietary model, as described above. In various embodiments, theanomalies may be magnified for visual comparison between the actualvalues and the predicted values. A relative strength of each detectedanomaly may also be indicated (e.g., by indicators having differentsizes or different colors or different shapes etc. to visuallydistinguish them based on strength level).

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. Modules may constitute eithersoftware modules (e.g., code embodied on a machine-readable medium or ina transmission signal) or hardware modules. A hardware module is atangible unit capable of performing certain operations and may beconfigured or arranged in a certain manner. In example embodiments, oneor more computer systems (e.g., a standalone, client or server computersystem) or one or more hardware modules of a computer system (e.g., aprocessor or a group of processors) may be configured by software (e.g.,an application or application portion) as a hardware module thatoperates to perform certain operations as described herein.

In various embodiments, a hardware module may be implementedmechanically or electronically. For example, a hardware module maycomprise dedicated circuitry or logic that is permanently configured(e.g., as a special-purpose processor, such as a field programmable gatearray (FPGA) or an application-specific integrated circuit (ASIC)) toperform certain operations. A hardware module may also compriseprogrammable logic or circuitry (e.g., as encompassed within ageneral-purpose processor or other programmable processor) that istemporarily configured by software to perform certain operations. Itwill be appreciated that the decision to implement a hardware modulemechanically, in dedicated and permanently configured circuitry, or intemporarily configured circuitry (e.g., configured by software) may bedriven by cost and time considerations.

Accordingly, the term “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired) or temporarilyconfigured (e.g., programmed) to operate in a certain manner and/or toperform certain operations described herein. Considering embodiments inwhich hardware modules are temporarily configured (e.g., programmed),each of the hardware modules need not be configured or instantiated atany one instance in time. For example, where the hardware modulescomprise a general-purpose processor configured using software, thegeneral-purpose processor may be configured as respective differenthardware modules at different times. Software may accordingly configurea processor, for example, to constitute a particular hardware module atone instance of time and to constitute a different hardware module at adifferent instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multipleof such hardware modules exist contemporaneously, communications may beachieved through signal transmission (e.g., over appropriate circuitsand buses) that connect the hardware modules. In embodiments in whichmultiple hardware modules are configured or instantiated at differenttimes, communications between such hardware modules may be achieved, forexample, through the storage and retrieval of information in memorystructures to which the multiple hardware modules have access. Forexample, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods described herein may be at least partiallyprocessor-implemented. For example, at least some of the operations of amethod may be performed by one or more processors orprocessor-implemented modules. The performance of certain of theoperations may be distributed among the one or more processors, not onlyresiding within a single machine, but deployed across a number ofmachines. In some example embodiments, the processor or processors maybe located in a single location (e.g., within a home environment, anoffice environment or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The one or more processors may also operate to support performance ofthe relevant operations in a “cloud computing” environment or as a“software as a service” (SaaS). For example, at least some of theoperations may be performed by a group of computers (as examples ofmachines including processors), these operations being accessible via anetwork (e.g., the network 104 of FIG. 1) and via one or moreappropriate interfaces (e.g., APIs).

Example embodiments may be implemented in digital electronic circuitry,or in computer hardware, firmware, software, or in combinations of them.Example embodiments may be implemented using a computer program product,e.g., a computer program tangibly embodied in an information carrier,e.g., in a machine-readable medium for execution by, or to control theoperation of, data processing apparatus, e.g., a programmable processor,a computer, or multiple computers.

A computer program can be written in any form of programming language,including compiled or interpreted languages, and it can be deployed inany form, including as a stand-alone program or as a module, subroutine,or other unit suitable for use in a computing environment. A computerprogram can be deployed to be executed on one computer or on multiplecomputers at one site or distributed across multiple sites andinterconnected by a communication network.

In example embodiments, operations may be performed by one or moreprogrammable processors executing a computer program to performfunctions by operating on input data and generating output. Methodoperations can also be performed by, and apparatus of exampleembodiments may be implemented as, special purpose logic circuitry(e.g., a FPGA or an ASIC).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. Inembodiments deploying a programmable computing system, it will beappreciated that both hardware and software architectures requireconsideration. Specifically, it will be appreciated that the choice ofwhether to implement certain functionality in permanently configuredhardware (e.g., an ASIC), in temporarily configured hardware (e.g., acombination of software and a programmable processor), or a combinationof permanently and temporarily configured hardware may be a designchoice. Below are set out hardware (e.g., machine) and softwarearchitectures that may be deployed, in various example embodiments.

FIG. 7 is a block diagram of machine in the example form of a computersystem 1800 within which instructions for causing the machine to performany one or more of the methodologies discussed herein may be executed.In alternative embodiments, the machine operates as a standalone deviceor may be connected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of a server or aclient machine in server-client network environment, or as a peermachine in a peer-to-peer (or distributed) network environment. Themachine may be a personal computer (PC), a tablet PC, a set-top box(STB), a Personal Digital Assistant (PDA), a cellular telephone, a webappliance, a network router, switch or bridge, or any machine capable ofexecuting instructions (sequential or otherwise) that specify actions tobe taken by that machine. Further, while only a single machine isillustrated, the term “machine” shall also be taken to include anycollection of machines that individually or jointly execute a set (ormultiple sets) of instructions to perform any one or more of themethodologies discussed herein.

The example computer system 1800 includes a processor 1802 (e.g., acentral processing unit (CPU), a graphics processing unit (GPU) orboth), a main memory 1804 and a static memory 1806, which communicatewith each other via a bus 1808. The computer system 1800 may furtherinclude a video display unit 1810 (e.g., a liquid crystal display (LCD)or a cathode ray tube (CRT)). The computer system 1800 also includes analphanumeric input device 1812 (e.g., a keyboard), a user interface (UI)navigation (or cursor control) device 1814 (e.g., a mouse), a storageunit 1816, a signal generation device 1818 (e.g., a speaker) and anetwork interface device 1820.

The storage unit 1816 includes a machine-readable medium 1822 on whichis stored one or more sets of data structures and instructions 1824(e.g., software) embodying or utilized by any one or more of themethodologies or functions described herein. The instructions 1824 mayalso reside, completely or at least partially, within the main memory1804 and/or within the processor 1802 during execution thereof by thecomputer system 1800, the main memory 1804 and the processor 1802 alsoconstituting machine-readable media. The instructions 1824 may alsoreside, completely or at least partially, within the static memory 1806.

While the machine-readable medium 1822 is shown in an example embodimentto be a single medium, the term “machine-readable medium” may include asingle medium or multiple media (e.g., a centralized or distributeddatabase, and/or associated caches and servers) that store the one ormore instructions 1824 or data structures. The term “machine-readablemedium” shall also be taken to include any tangible medium that iscapable of storing, encoding or carrying instructions for execution bythe machine and that cause the machine to perform any one or more of themethodologies of the present embodiments, or that is capable of storing,encoding or carrying data structures utilized by or associated with suchinstructions. The term “machine-readable medium” shall accordingly betaken to include, but not be limited to, solid-state memories, andoptical and magnetic media. Specific examples of machine-readable mediainclude non-volatile memory, including by way of example semiconductormemory devices, e.g., Erasable Programmable Read-Only Memory (EPROM),Electrically Erasable Programmable Read-Only Memory (EEPROM), and flashmemory devices; magnetic disks such as internal hard disks and removabledisks; magneto-optical disks; and compact disc-read-only memory (CD-ROM)and digital versatile disc (or digital video disc) read-only memory(DVD-ROM) disks.

Accordingly, a “tangible machine-readable medium” may refer to a singlestorage apparatus or device, as well as “cloud-based” storage systems orstorage networks that include multiple storage apparatus or devices.Furthermore, the tangible machine-readable medium is non-transitory inthat it does not embody a propagating signal. However, labeling thetangible machine-readable medium as “non-transitory” should not beconstrued to mean that the medium is incapable of movement—the mediumshould be considered as being transportable from one physical locationto another. Additionally, since the machine-readable medium is tangible,the medium may be considered to be a machine-readable device.

The instructions 1824 may further be transmitted or received over acommunications network 1826 using a transmission medium. Theinstructions 1824 may be transmitted using the network interface device1820 and any one of a number of well-known transfer protocols (e.g.,HTTP). Examples of communication networks include a LAN, a WAN, theInternet, mobile telephone networks, POTS networks, and wireless datanetworks (e.g., WiFi and WiMax networks). The term “transmission medium”shall be taken to include any intangible medium capable of storing,encoding or carrying instructions for execution by the machine, andincludes digital or analog communications signals or other intangiblemedia to facilitate communication of such software. The network 1826 maybe one of the networks 104.

Although an embodiment has been described with reference to specificexample embodiments, it will be evident that various modifications andchanges may be made to these embodiments without departing from thebroader spirit and scope of the present disclosure. Accordingly, thespecification and drawings are to be regarded in an illustrative ratherthan a restrictive sense. The accompanying drawings that form a parthereof, show by way of illustration, and not of limitation, specificembodiments in which the subject matter may be practiced. Theembodiments illustrated are described in sufficient detail to enablethose skilled in the art to practice the teachings disclosed herein.Other embodiments may be utilized and derived therefrom, such thatstructural and logical substitutions and changes may be made withoutdeparting from the scope of this disclosure. This Detailed Description,therefore, is not to be taken in a limiting sense, and the scope ofvarious embodiments is defined only by the appended claims, along withthe full range of equivalents to which such claims are entitled.

Such embodiments of the inventive subject matter may be referred toherein, individually and/or collectively, by the term “invention” merelyfor convenience and without intending to voluntarily limit the scope ofthis application to any single invention or inventive concept if morethan one is in fact disclosed. Thus, although specific embodiments havebeen illustrated and described herein, it should be appreciated that anyarrangement calculated to achieve the same purpose may be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent to those of skill in theart upon reviewing the above description.

What is claimed is:
 1. A system comprising: one or more computerprocessors; one or more computer memories; one or more modulesincorporated into the one or more computer memories, the one or moremodules configuring the one or more computer processors to performoperations, the operations comprising: extracting a training time seriescorresponding to a process from an initial time series corresponding tothe process; modifying outlier data points in the training time seriesbased on predetermined acceptability criteria; training a plurality ofprediction methods using the training time series; receiving an actualdata point corresponding to the initial time series; using the pluralityof prediction methods to determine a set of predicted data pointscorresponding to the actual data point of the initial time series;determining whether the actual data point is anomalous based on acalculation of whether each of the set of predicted data points isstatistically different from the actual data point; and receiving anadditional actual data point corresponding to the initial time seriesand extracting an additional training time series from the initial timeseries based on the additional actual data point.
 2. The system of claim1, wherein the calculation of whether each of the set of predicted datapoints is statistically different from the actual data point includes adetermination that the Mahalanobis distance between the prediction errorand the fitted multivariate normal joint probability distribution ofeach of the set of predicted data points is within a specified range. 3.The system of claim 1, wherein the additional actual data pointcorresponds to the initial time series and the operations furthercomprise extracting an additional training time series having the lengthoffset by an additional index prior to a last data point of the initialtime series the additional index reflecting a relative position of theactual data point to the additional actual data point.
 4. The system ofclaim 1, further comprising selecting the combination of each of theplurality of prediction methods to minimize a number of false anomalydetections.
 5. The system of claim 1, further comprising representingthe determination of whether the actual data point is anomalous in agraphical user interface, the representing including providing astrength of the determination.
 6. The system of claim 5, wherein thestrength of the determination is based on a number of the plurality ofprediction methods that indicate an anomaly with respect to the datapoint.
 7. The system of claim 1, wherein the training time seriesrepresents a window of the initial time series that is recent inrelation to the actual data point.
 8. A method comprising: extracting atraining time series corresponding to a process from an initial timeseries corresponding to the process; modifying outlier data points inthe training time series based on predetermined acceptability criteria;training a plurality of prediction methods using the training timeseries; receiving an actual data point corresponding to the initial timeseries; using the plurality of prediction methods to determine a set ofpredicted data points corresponding to the actual data point of theinitial time series; determining whether the actual data point isanomalous based on a calculation of whether each of the set of predicteddata points is statistically different from the actual data point; andreceiving an additional actual data point corresponding to the initialtime series and extracting an additional training time series from theinitial time series based on the additional actual data point.
 9. Themethod of claim 8, wherein the calculation of whether each of the set ofpredicted data points is statistically different from the actual datapoint includes a determination that the Mahalanobis distance between theprediction error and the fitted multivariate normal joint probabilitydistribution of each of the set of predicted data points is within aspecified range.
 10. The method of claim 8, wherein additional actualdata point corresponds to the initial time series and the method furthercomprises extracting an additional training time series having thelength offset by an additional index prior to a last data point of theinitial time series the additional index reflecting a relative positionof the actual data point to the additional actual data point.
 11. Themethod of claim 8, further comprising selecting the combination of eachof the plurality of prediction methods to minimize a number of falseanomaly detections.
 12. The method of claim 8, further comprisingrepresenting the determination of whether the actual data point isanomalous in a graphical user interface, the representing includingproviding a strength of the determination.
 13. The method of claim 12,wherein the strength of the determination is based on a number of theplurality of prediction methods that indicate an anomaly with respect tothe data point.
 14. The method of claim 8, wherein the training timeseries represents a window of the initial time series that is recent inrelation to the actual data point.
 15. A non-transitory machine readablemedium comprising a set of instructions that, when executed by aprocessor, causes the processor to perform operations, the operationscomprising: extracting a training time series corresponding to a processfrom an initial time series corresponding to the process; modifyingoutlier data points in the training time series based on predeterminedacceptability criteria; training a plurality of prediction methods usingthe training time series; receiving an actual data point correspondingto the initial time series; using the plurality of prediction methods todetermine a set of predicted data points corresponding to the actualdata point of the initial time series; determining whether the actualdata point is anomalous based on a calculation of whether each of theset of predicted data points is statistically different from the actualdata point; and receiving an additional actual data point correspondingto the initial time series and extracting an additional training timeseries from the initial time series based on the additional actual datapoint.
 16. The non-transitory machine readable medium of claim 15,wherein the calculation of whether each of the set of predicted datapoints is statistically different from the actual data point includes adetermination that the Mahalanobis distance between the prediction errorand the fitted multivariate normal joint probability distribution ofeach of the set of predicted data points is within a specified range.17. The non-transitory machine readable medium of claim 15, wherein theadditional actual data point corresponds to the initial time series andthe operations further comprise extracting an additional training timeseries having the length offset by an additional index prior to a lastdata point of the initial time series the additional index reflecting arelative position of the actual data point to the additional actual datapoint.
 18. The non-transitory machine readable medium of claim 15, theoperations further comprising selecting the combination of each of theplurality of prediction methods to minimize a number of false anomalydetections.
 19. The non-transitory machine readable medium of claim 15,the operations further comprising representing the determination ofwhether the actual data point is anomalous in a graphical userinterface, the representing including providing a strength of thedetermination.
 20. The non-transitory machine readable medium of claim19, wherein the strength of the determination is based on a number ofthe plurality of prediction methods that indicate an anomaly withrespect to the data point.